Entropy and Huffman Codes

We will show that • the entropy for a random variable gives a lower bound on the number of bits needed per character for a binary coding • Huffman codes are optimal in the average number of bits used per character among binary codes • the average bits per character used by Huffman codes is close to the entropy of the underlying random variable • one can get arbitrarily close to the entropy of a random variable in terms of bits per character if blocks of several characters are encoded at a time First, some basic notation, some of which is associated to a discrete random variable X. We let